feat(v1): task-authored interception of model exchanges (@vf.intercept)#2068
feat(v1): task-authored interception of model exchanges (@vf.intercept)#2068xeophon wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 601abece6a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def stream_events(self, raw: dict) -> list[bytes]: | ||
| # The terminal event carrying the full object, then the DONE sentinel. | ||
| completed = {"type": "response.completed", "response": raw} |
There was a problem hiding this comment.
Preserve Responses terminal failure state
When a streamed Responses call is mutated by any response-side interceptor, this replay path always emits response.completed, even if the raw response being replayed has status == "incomplete" or "failed" (the parser above explicitly treats response.incomplete and response.failed as terminal events). In those intercepted streaming cases, clients such as Codex will see a successful completion event instead of the original truncation/failure signal, so they can continue as if the turn completed normally while the trace records a length/failed response.
Useful? React with 👍 / 👎.
| if isinstance(item, dict) and _matches( | ||
| matcher, item.get("type"), item.get("name") | ||
| ): |
There was a problem hiding this comment.
Do not strip model-invoked Responses tools
For Responses output this matches on both type and name, so strip_provider_tools(exchange, "web_search") will also delete a normal model-invoked function_call item whose function is named web_search, not just provider-side items like web_search_call. In a task that exposes its own function/MCP tool with a matching name, the model's legitimate tool call is removed before the harness can execute it, and the turn is re-derived as if no tool call was made.
Useful? React with 👍 / 👎.
Summary
Adds task-authored interception of every model exchange to v1: an
@vf.interceptdecorator (mirroring@vf.stop/@vf.reward) that can block, rewrite, strip, or terminate exchanges — for all harnesses at once, at the one choke point all model traffic funnels through (the interception server).vf.Messagewhose text answers the model instead, so it can do something else. Provider-side tool output (Codex/Claude web search) can be stripped from the response. Streaming responses are buffered when handlers are registered, then re-emitted (byte-identical when untouched).vf.Terminate(reason=..., reward=-1.0)ends the rollout on the spot and records the reward — direct punishment for reward hacking.trace.interceptions); in-place mutations ofexchange.raware auto-detected and logged, so handlers only ever return a message, aTerminate, or nothing.Prebuilt helpers (
verifiers/v1/intercepts/, importable like judges)match_tool/match_tool_calls— fuzzy tool-name matching, so one pattern (e.g.web_search) covers Claude'sweb_search_20250305and Codex'sweb_search_previewjudge_tools(exchange, judge=...) -> bool— an LLM judge (ToolCallJudge, or any customJudgesubclass with its own prompt) vets each tool call; the handler decides what a rejection meansstrip_provider_tools(exchange, ...) -> list[str]— strips provider-side tools in both directions and records what was removedShowcase environment
environments/intercept_v1/demonstrates all four behaviors on a 4-row inline dataset: blocking git commands (the model gets an error and continues), a custom judge gating tool calls, provider-side web-search stripping, and terminating with reward −1 on reward-hacking attempts.Verification
uv run pytest tests/v1 tests/test_imports.py ...— 93 passed (e2e skips need API keys); ruff, pre-commit, andtycleanInterceptionServer: blocked tool calls never reach the harness (HTTP + SSE, with passthrough controls), strips/terminates/replays behave, and streaming is byte-identical when no handlers are registeredNote
High Risk
Changes core interception-server request/response paths and streaming behavior for all harnesses; incorrect handling could block legitimate tool use or alter provider traffic silently.
Overview
Introduces task-authored interception at the single model-traffic choke point (the interception server), via a new
@vf.interceptdecorator parallel to@vf.stop/@vf.reward.Handlers run on every exchange in priority order on both request (mutate wire JSON before upstream) and response (before the turn commits). They can return a
Messageto block tool calls and answer the model in text,Terminateto end the rollout (optional negative reward), or mutateexchange.rawin place (rewrites are auto-logged). Actions are stored ontrace.interceptions.The interception server wires this through
RolloutSession.run_intercepts: request-side termination refuses the turn; response-side blocking drops tool calls via dialect-aware wire helpers. Streaming buffers the full SSE when handlers exist, then replays original chunks ordialect.stream_eventsafter mutations. Parsers now attachresponse.rawfor in-place edits.Prebuilt helpers (
match_tool,judge_tools,strip_provider_tools) cover fuzzy tool names, LLM judge gating, and stripping provider-side web search across chat / Anthropic / Responses shapes.Adds
intercept-v1(four demo rows: terminate on answer-file reads, block git, judge gate, strip web search) and registers it in the repoexamples/uvworkspace.Reviewed by Cursor Bugbot for commit 601abec. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add
@vf.interceptdecorator for task-authored interception of model exchanges@vf.interceptdecorator in decorators.py that tasks can use to register handler functions with an optional priority, discovered at rollout time and passed intoRolloutSession.RolloutSession.run_interceptsin session.py runs handlers in priority order on each request/response exchange; handlers can silently mutate the wire payload, return aMessageto block a tool call, or returnTerminateto end the rollout with an optional reward.Response.rawand implementsstream_eventsto re-synthesize SSE from a mutated raw object for replay.match_tool,TOOL_SYNONYMS), provider tool stripping (strip_provider_tools), and an LLM-based tool-call gate (judge_tools/ToolCallJudge).Macroscope summarized 601abec.